Infrared camera systems and methods for dual sensor applications

ABSTRACT

Systems and methods disclosed herein provide for infrared camera systems and methods for dual sensor applications. For example, in one embodiment, an enhanced vision system comprises an image capture component having a visible light sensor to capture visible light images and an infrared sensor to capture infrared images. The system comprises a first control component adapted to provide a plurality of selectable processing modes to a user, receive a user input corresponding to a user selected processing mode, and generate a control signal indicative of the user selected processing mode, wherein the plurality of selectable processing modes includes a visible light only mode, infrared only mode, and a combined visible-infrared mode. The system comprises a processing component adapted to receive the generated control signal from the control component, process the captured visible light images and the captured infrared images according to the user selected processing mode, and generate processed images based on the processing mode selected by the user. The system comprises a display component adapted to display the processed images based on the processing mode selected by the user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is a continuation of U.S. patent applicationSer. No. 12/477,828 filed Jun. 3, 2009, which is incorporated byreference herein in its entirety.

TECHNICAL FIELD

The present disclosure relates to infrared imaging systems and, inparticular, to infrared camera systems and methods for dual sensorapplications.

BACKGROUND

Visible light cameras are utilized in a variety of imaging applicationsto capture color or monochrome images. For instance, visible lightcameras are typically utilized for daytime applications under ambientlight.

Infrared cameras are utilized in a variety of imaging applications tocapture thermal emission from objects as infrared images and, therefore,may not be dependent on ambient lighting. For instance, infrared camerasmay be utilized for nighttime applications to enhance visibility underlow-lighting conditions that typically affect cameras limited to thevisible spectral band.

However, there are several drawbacks for conventional nighttimeimplementation approaches for visible light cameras and infraredcameras. For instance, vehicle operators may have to alternate betweendisplays presenting images from visible light cameras and displayspresenting images from infrared cameras to establish a correlationbetween the visible-band and infrared-band representation of the sceneand to determine path obstructions and directions for safe passage. Thiscan be relatively difficult. Even with a split view on a single display,the operator has to constantly switch between each view to determinewhat in the image may be an obstacle to correlate the output fromvisible light cameras and infrared cameras in a multi-band camerasystem.

As a result, there is a need for improved display techniques forproviding visible light images and infrared images in an easily viewablemanner. There is also a need for improved visible light and infraredcamera processing techniques for various applications including, e.g.,nighttime applications.

SUMMARY

Systems and methods disclosed herein, in accordance with one or moreembodiments, provide processing techniques and modes of operation fordual sensor imaging devices, such as dual sensor cameras, that may beadapted to provide enhanced vision capability. In one example, undercertain conditions, it is desirable for vehicles (e.g., aircraft,watercraft, land based vehicles, etc.) to use enhanced vision systems toaid a pilot in operating and navigating the vehicle. For instance, atnighttime, the use of a sensor sensitive to infrared radiation assistswith imaging in darkness. Under circumstances where visible lightsources exist, it is desirable for the pilot to see those lights with asensor sensitive to the visible light spectrum. Accordingly, embodimentsof the present disclosure provide an enhanced vision system forcombining captured images from infrared and visible wavelength sensorsfor processing applications requiring minimal user input. By combiningor blending image signals from a visible light sensor with image signalsfrom an infrared sensor, a combined or blended image may be generatedthat retains color information from the visible light sensor and showsinfrared luminance from the infrared sensor.

In accordance with an embodiment, an enhanced vision system includes atleast one sensor sensitive to light in the visible spectrum and at leastone sensor sensitive to theimal radiation in the infrared spectrum. Theenhanced vision system is adapted to operate in at least three modes,wherein a first mode displays an image signal from the visible sensor, asecond mode displays an image signal from the infrared sensor, and athird mode displays an image signal that is generated by combining theimage signal from the visible sensor and the image signal from theinfrared sensor. In one implementation, the combined image signal in thethird mode is generated by combining a luminance part of the imagesignal from the visible sensor with a luminance part from the imagesignal of the infrared sensor to generate the luminance part of thecombined image signal. Chrominance information in the combined imagesignal may be retained from the image signal of the visible sensor. Themodes of operation are user selectable, wherein a single user input(such as produced by turning a knob) controls the image signals in anyof the operating modes. In one aspect, the enhanced vision system mayautomatically select the mode of operation based on time of day. Inanother aspect, the enhanced vision system may automatically select themode of operation based on properties (e.g., SNR: signal-to-noise ratio)of the captured image signals in either or both bands (e.g., visiblelight band and/or infrared band).

In accordance with an embodiment, an enhanced vision system includes animage capture component having a visible light sensor to capture visiblelight images and an infrared sensor to capture infrared images. Theenhanced vision system includes a first control component adapted toprovide a plurality of selectable processing modes to a user, receive auser input corresponding to a user selected processing mode, andgenerate a control signal indicative of the user selected processingmode, wherein the plurality of selectable processing modes includes avisible light only mode, infrared only mode, and a combinedvisible-infrared mode. The enhanced vision system includes a processingcomponent adapted to receive the generated control signal from thecontrol component, process the captured visible light images and thecaptured infrared images according to the user selected processing mode,and generate processed images based on the processing mode selected bythe user. The enhanced vision system includes a display componentadapted to display the processed images based on the processing modeselected by the user.

The plurality of selectable processing modes includes the visible lightonly mode that causes the processing component to generate the outputimages from only the captured visible light images, wherein the visiblelight images comprise color information or monochrome information fromthe visible light wavelength spectrum. The plurality of selectableprocessing modes includes the infrared only mode that causes theprocessing component to generate the output images from only thecaptured infrared images, wherein the infrared images comprise thermalinformation from the infrared wavelength spectrum. The plurality ofselectable processing modes includes the combined visible-infrared modethat causes the processing component to generate the processed images bycombining a part of the captured visible light images with a part fromthe captured infrared images.

The enhanced vision system may include a memory component adapted tostore the captured visible light images, the captured infrared images,and the processed images. The enhanced vision system may include asecond control component adapted to provide a selectable controlparameter (ξ) to the user, receive a user input corresponding to a userselected control parameter (ξ), and generate a control signal indicativeof the user selected control parameter (ξ). The selectable controlparameter (ξ) is normalized to have a value in the range of 0 (zero)to 1. In the blended mode, a value of 1 causes the processing componentto generate the output image from only the captured visible lightimages, and wherein a value of 0 (zero) causes the processing componentto generate the output image from only the captured infrared images. Inblended mode, values between 0 (zero) and 1 causes the processingcomponent to generate the processed images from proportional parts ofboth the captured visible light images and the captured infrared images.The selectable control parameter (ξ) is adapted to affect proportions ofluminance values of the visible light images and the infrared images inthe processed images when the combined visible-infrared mode is selectedby the user.

The processed images comprise visible light only images when the visiblelight only mode is selected by the user, and wherein the processedimages comprise infrared only images when the infrared only mode isselected by the user, and wherein the processed images comprise combinedvisible-infrared images when the combined visible-infrared mode isselected by the user. The display component is adapted to display theprocessed images as visible light only images, infrared only images, andcombined visible-infrared images having portions of both the visiblelight only images and the infrared only images.

The captured visible light images comprise a luminance (Y) part and achrominance (CrCb) part, and wherein the captured infrared imagescomprise only a luminance (Y) part, and wherein the processing componentis adapted to extract the luminance (Y) part and the chrominance (CrCb)part from the captured visible light images, extract the luminance (Y)part from the captured infrared images, scale the luminance (Y) parts,and merge the luminance (Y) parts and the chrominance (CrCb) part togenerate the processed image based on the processing mode selected bythe user.

In accordance with an embodiment, a method for processing imagescomprises capturing a visible light image and an infrared image andproviding a plurality of selectable processing modes to a user, whereinthe plurality of selectable processing modes includes a visible lightonly mode, infrared only mode, and a combined visible-infrared mode. Themethod includes receiving a user input corresponding to a user selectedprocessing mode, processing the captured visible light images and thecaptured infrared images according to the user selected processing mode,generating processed images, and displaying the processed images. Theprocessed images may comprise visible light only images when the visiblelight only mode is selected by the user. The processed images maycomprise infrared only images when the infrared only mode is selected bythe user. The processed images may comprise combined visible-infraredimages when the combined visible-infrared mode is selected by the user.

The scope of the disclosure is defined by the claims, which areincorporated into this section by reference. A more completeunderstanding of embodiments of the present disclosure may be affordedto those skilled in the art, as well as a realization of additionaladvantages thereof, by a consideration of the following detaileddescription of one or more embodiments. Reference may be made to theappended sheets of drawings that may first be described briefly.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1B show block diagrams illustrating various infrared imagingsystems for capturing and processing infrared images, in accordance withvarious embodiments of the present disclosure.

FIGS. 1C-1D show block diagrams illustrating various configurations forthe infrared imaging systems of FIG. 1B, in accordance with variousembodiments of the present disclosure.

FIGS. 1E-1F show schematic diagrams illustrating a marine application ofthe infrared imaging systems of FIG. 1B, in accordance with variousembodiments of the present disclosure.

FIG. 2 shows a block diagram illustrating a method for capturing andprocessing infrared images, in accordance with an embodiment of thepresent disclosure.

FIGS. 3A-3F show block diagrams illustrating infrared processingtechniques, in accordance with various embodiments of the presentdisclosure.

FIG. 4 shows a block diagram illustrating an overview of infraredprocessing techniques, in accordance with various embodiments of thepresent disclosure.

FIG. 5 shows a block diagram illustrating a control component of theinfrared imaging system for selecting between different modes ofoperation, in accordance with an embodiment of the present disclosure.

FIG. 6 shows a schematic diagram illustrating an embodiment of an imagecapture component of infrared imaging systems, in accordance with anembodiment of the present disclosure.

FIG. 7 shows a block diagram illustrating an embodiment of a method formonitoring image data of the infrared imaging systems, in accordancewith an embodiment of the present disclosure.

FIG. 8 shows a block diagram of a method for implementing dual sensorapplications in an enhanced vision system, in accordance withembodiments of the present disclosure.

FIG. 9 shows a block diagram illustrating a method for implementing oneor more enhanced vision modes of operation and infrared processingtechniques related thereto, in accordance with embodiments of thepresent disclosure.

FIG. 10 shows a block diagram illustrating a control component of anenhanced vision system for selecting between one or more enhanced visionmodes of operation, in accordance with embodiments of the presentdisclosure.

FIG. 11A shows a graph diagram of a per pixel blending adjustment, inaccordance with an embodiment of the present disclosure.

FIG. 11B shows a graph diagram of a per pixel brightness adjustment, inaccordance with an embodiment of the present disclosure.

Embodiments of the present disclosure and their advantages are bestunderstood by referring to the detailed description that follows. Itshould be appreciated that like reference numerals are used to identifylike elements illustrated in one or more of the figures.

DETAILED DESCRIPTION

In accordance with an embodiment of the present disclosure, FIG. 1Ashows a block diagram illustrating an infrared imaging system 100A forcapturing and processing infrared images. Infrared imaging system 100Acomprises a processing component 110, a memory component 120, an imagecapture component 130, a display component 140, a control component 150,and optionally a sensing component 160.

In various implementations, infrared imaging system 100A may representan infrared imaging device, such as an infrared camera, to capture oneor more images, such as an image 170. Infrared imaging system 100A mayrepresent any type of infrared camera, which for example detectsinfrared radiation and provides representative data (e.g., one or moresnapshots or video infrared images). For example, infrared imagingsystem 100A may represent an infrared camera that is directed to thenear, middle, and/or far infrared spectrums. Infrared imaging system100A may comprise a portable device and may be incorporated, forexample, into a vehicle (e.g., a naval vehicle, a land-based vehicle, anaircraft, or a spacecraft) or a non-mobile installation requiringinfrared images to be stored and/or displayed.

Processing component 110 comprises, in one embodiment, a microprocessor,a single-core processor, a multi-core processor, a microcontroller, alogic device (e.g., a programmable logic device configured to performprocessing functions), a digital signal processing (DSP) device, or someother type of generally known processor. Processing component 110 isadapted to interface and communicate with components 120, 130, 140, 150and 160 to perform method and processing steps as described herein.Processing component 110 may comprise one or more mode modules 112A-112Nfor operating in one or more modes of operation. In one implementation,mode modules 112A-112N define preset display functions that may beembedded in processing component 110 or stored on memory component 120for access and execution by processing component 110. Moreover,processing component 110 may be adapted to perform various other typesof image processing algorithms.

In various implementations, it should be appreciated that each of modemodules 112A-112N may be integrated in software and/or hardware as partof processing component 110, or code (e.g., software or configurationdata) for each of the modes of operation associated with each modemodule 112A-112N, which may be stored in memory component 120.Embodiments of mode modules 112A-112N (i.e., modes of operation) may bestored by a separate computer-readable medium (e.g., a memory, such as ahard drive, a compact disk, a digital video disk, or a flash memory) tobe executed by a computer (e.g., a logic or processor-based system) toperform various methods. In one example, the computer-readable mediummay be portable and/or located separate from infrared imaging system100A, with stored mode modules 112A-112N provided to infrared imagingsystem 100A by coupling the computer-readable medium to infrared imagingsystem 100A and/or by infrared imaging system 100A downloading (e.g.,via a wired or wireless link) the mode modules 112A-112N from thecomputer-readable medium. Mode modules 112A-112N provide for improvedinfrared camera processing techniques for real time applications,wherein a user or operator may change the mode while viewing an image ondisplay component 140.

Memory component 120 comprises, in one embodiment, one or more memorydevices to store data and information. The one or more memory devicesmay comprise various types of memory including volatile and non-volatilememory devices, such as RAM (Random Access Memory), ROM (Read-OnlyMemory), EEPROM (Electrically-Erasable Read-Only Memory), flash memory,etc. Processing component 110 is adapted to execute software stored inmemory component 120 to perform methods, processes, and modes ofoperations.

Image capture component 130 comprises, in one embodiment, one or moreinfrared sensors (e.g., any type of infrared detector, such as a focalplane array) for capturing infrared image signals representative of animage, such as image 170. In one implementation, the infrared sensors ofimage capture component 130 provide for representing (e.g., converting)a captured image signal of image 170 as digital data (e.g., via ananalog-to-digital converter included as part of the infrared sensor orseparate from the infrared sensor as part of infrared imaging system100A). Processing component 110 may be adapted to receive the infraredimage signals from image capture component 130, process the infraredimage signals (e.g., provide processed image data), store the infraredimage signals or image data in memory component 120, and/or retrievestored infrared image signals from memory component 120. Processingcomponent 110 may be adapted to process infrared image signals stored inmemory component 120 to provide image data (e.g., captured and/orprocessed infrared image data) to display component 140 for viewing by auser.

Display component 140 comprises, in one embodiment, an image displaydevice (e.g., a liquid crystal display (LCD)) or various other types ofgenerally known video displays or monitors. Processing component 110 maybe adapted to display image data and information on display component140. Processing component 110 may also be adapted to retrieve image dataand information from memory component 120 and display any retrievedimage data and information on display component 140. Display component140 may comprise display electronics, which may be utilized byprocessing component 110 to display image data and information (e.g.,infrared images). Display component 140 may receive image data andinformation directly from image capture component 130 via processingcomponent 110, or the image data and information may be transferred frommemory component 120 via processing component 110. In oneimplementation, processing component 110 may initially process acaptured image and present a processed image in one mode, correspondingto mode modules 112A-112N, and then upon user input to control component150, processing component 110 may switch the current mode to a differentmode for viewing the processed image on display component 140 in thedifferent mode. This switching may be referred to as applying theinfrared camera processing techniques of mode modules 112A-112N for realtime applications, wherein a user or operator may change the mode whileviewing an image on display component 140 based on user input to controlcomponent 150.

Control component 150 comprises, in one embodiment, a user input and/orinterface device having one or more user actuated components, such asone or more push buttons, slide bars, rotatable knobs or a keyboard,that are adapted to generate one or more user actuated input controlsignals. Control component 150 may be adapted to be integrated as partof display component 140 to function as both a user input device and adisplay device, such as, for example, a touch screen device adapted toreceive input signals from a user touching different parts of thedisplay screen. Processing component 110 may be adapted to sense controlinput signals from control component 150 and respond to any sensedcontrol input signals received therefrom. Processing component 110 maybe adapted to interpret the control input signal as a value, which maybe described in greater detail herein.

Control component 150 may comprise, in one embodiment, a control panelunit 500 (e.g., a wired or wireless handheld control unit) having one ormore push buttons adapted to interface with a user and receive userinput control values, as shown in FIG. 5 and further described herein.In various implementations, one or more push buttons of control panelunit 500 may be utilized to select between the various modes ofoperation as described herein in reference to FIGS. 2-4. For example,only one push button may be implemented and which is used by theoperator to cycle through the various modes of operation (e.g., nightdocking, man overboard, night cruising, day cruising, hazy conditions,and shoreline), with the selected mode indicated on the displaycomponent 140. In various other implementations, it should beappreciated that control panel unit 500 may be adapted to include one ormore other push buttons to provide various other control functions ofinfrared imaging system 100A, such as auto-focus, menu enable andselection, field of view (FoV), brightness, contrast, gain, offset,spatial, temporal, and/or various other features and/or parameters. Inanother implementation, a variable gain value may be adjusted by theuser or operator based on a selected mode of operation.

In another embodiment, control component 150 may comprise a graphicaluser interface (GUI), which may be integrated as part of displaycomponent 140 (e.g., a user actuated touch screen), having one or moreimages of, for example, push buttons adapted to interface with a userand receive user input control values.

Optional sensing component 160 comprises, in one embodiment, one or morevarious types of sensors, including environmental sensors, dependingupon the desired application or implementation requirements, whichprovide information to processing component 110. Processing component110 may be adapted to communicate with sensing component 160 (e.g., byreceiving sensor information from sensing component 160) and with imagecapture component 130 (e.g., by receiving data from image capturecomponent 130 and providing and/or receiving command, control or otherinformation to and/or from other components of infrared imaging system100A).

In various implementations, optional sensing component 160 may providedata and information relating to environmental conditions, such asoutside temperature, lighting conditions (e.g., day, night, dusk, and/ordawn), humidity level, specific weather conditions (e.g., sun, rain,and/or snow), distance (e.g., laser rangefinder), and/or whether atunnel, a covered dock, or that some type of enclosure has been enteredor exited. Optional sensing component 160 may represent conventionalsensors as would be known by one skilled in the art for monitoringvarious conditions (e.g., environmental conditions) that may have anaffect (e.g., on the image appearance) on the data provided by imagecapture component 130.

In some embodiments, optional sensing component 160 may comprise one ormore devices adapted to relay information to processing component 110via wireless communication. For example, sensing component 160 may beadapted to receive information from a satellite, through a localbroadcast (e.g., radio frequency) transmission, through a mobile orcellular network and/or through information beacons in an infrastructure(e.g., a transportation or highway information beacon infrastructure) orvarious other wired or wireless techniques.

In various embodiments, components of image capturing system 100A may becombined and/or implemented or not, as desired or depending upon theapplication or requirements, with image capturing system 100Arepresenting various functional blocks of a system. For example,processing component 110 may be combined with memory component 120,image capture component 130, display component 140 and/or sensingcomponent 160.

In another example, processing component 110 may be combined with imagecapture component 130 with only certain functions of processingcomponent 110 performed by circuitry (e.g., a processor, amicroprocessor, a microcontroller, a logic device, etc.) within imagecapture component 130. In still another example, control component 150may be combined with one or more other components or be remotelyconnected to at least one other component, such as processing component110, via a control wire to as to provide control signals thereto.

In accordance with another embodiment of the present disclosure, FIG. 1Bshows a block diagram illustrating an infrared imaging system 100B forcapturing and processing infrared images. Infrared imaging system 100Bcomprises, in one embodiment, a processing component 110, an interfacecomponent 118, a memory component 120, one or more image capturecomponents 130A-130N, a display component 140, a control component 150,and optionally a sensing component 160. It should be appreciated thatvarious components of infrared imaging system 100B of FIG. 1B may besimilar in function and scope to components of infrared imaging system100A of FIG. 1A, and any differences between the systems 100A, 100B aredescribed in greater detail herein.

In various implementations, infrared imaging system 100B may representone or more infrared imaging devices, such as one or more infraredcameras, to capture images, such as images 170A-170N. In general,infrared imaging system 100B may utilize a plurality of infraredcameras, which for example detect infrared radiation and providerepresentative data (e.g., one or more snapshots or video infraredimages). For example, infrared imaging system 100B may include one ormore infrared cameras that are directed to the near, middle, and/or farinfrared spectrums. As discussed further herein, infrared imaging system100B may be incorporated, for example, into a vehicle (e.g., a navalvehicle or other type of watercraft, a land-based vehicle, an aircraft,or a spacecraft) or a non-mobile installation requiring infrared imagesto be stored and/or displayed.

Processing component 110 is adapted to interface and communicate with aplurality of components including components 118, 120, 130A-130N, 140,150, and/or 160 of system 100B to perform method and processing steps asdescribed herein. Processing component 110 may comprise one or more modemodules 112A-112N for operating in one or more modes of operation, whichis described in greater detail herein. Processing component 110 may beadapted to perform various other types of image processing algorithms ina manner as described herein.

Interface component 118 comprises, in one embodiment, a communicationdevice (e.g., modem, router, switch, hub, or Ethernet card) that allowscommunication between each image capture component 130A-130N andprocessing component 110. As such, processing component 110 is adaptedto receive infrared image signals from each image capture component130A-130N via interface component 118.

Each image capture component 130A-130N (where “N” represents any desirednumber) comprises, in various embodiments, one or more infrared sensors(e.g., any type of infrared detector, such as a focal plane array, orany type of infrared camera, such as infrared imaging system 100A) forcapturing infrared image signals representative of an image, such as oneor more images 170A-170N. In one implementation, the infrared sensors ofimage capture component 130A provide for representing (e.g., converting)a captured image signal of, for example, image 170A as digital data(e.g., via an analog-to-digital converter included as part of theinfrared sensor or separate from the infrared sensor as part of infraredimaging system 100B). As such, processing component 110 may be adaptedto receive the infrared image signals from each image capture component130A-130N via interface component 118, process the infrared imagesignals (e.g., to provide processed image data or the processed imagedata may be provided by each image capture component 130A-130N), storethe infrared image signals or image data in memory component 120, and/orretrieve stored infrared image signals from memory component 120.Processing component 110 may be adapted to process infrared imagesignals stored in memory component 120 to provide image data (e.g.,captured and/or processed infrared image data) to display component 140(e.g., one or more displays) for viewing by a user.

In one implementation, referring to FIG. 6, each image capture component130A-130N may comprise one or more components, including a first cameracomponent 132, a second camera component 134, and/or a searchlightcomponent 136. In one embodiment as shown in FIG. 6, first cameracomponent 132 is adapted to capture infrared images, second cameracomponent 134 is adapted to capture color images in a visible lightspectrum, and searchlight component 136 is adapted to provide a beam oflight to a position within an image boundary of the one or more images170 (e.g., within a field of view of first camera component 132 and/orsecond camera component 134).

FIG. 1C shows a top-view of infrared imaging system 100B having aplurality of image capture components 130A-130D (e.g., infrared cameras)mounted to a watercraft 180 in accordance with an embodiment of thepresent disclosure. In various implementations, image capture components130A-130D may comprise any type of infrared camera (e.g., infrareddetector device) adapted to capture one or more infrared images.Watercraft 180 may represent any type of watercraft (e.g., a boat,yacht, ship, cruise ship, tanker, commercial vessel, military vessel,etc.).

As shown in FIG. 1C, a plurality of image capture components 130A-130Dmay be mounted in a configuration at different positions on watercraft180 in a manner so as to provide one or more fields of view aroundwatercraft 180. In various implementations, an image capture component130A may be mounted to provide a field of view ahead of or around a bow182 (e.g., forward or fore part) of watercraft 180. As further shown, animage capture component 130B may be mounted to provide a field of viewto the side of or around a port 184 (e.g., left side when facing bow182) of watercraft 180. As further shown, an image capture component130C may be mounted to provide a field of view to the side of or arounda starboard 186 (e.g., right side when facing bow 182) of watercraft180. As further shown, an image capture component 130D may be mounted toprovide a field of view behind of or around a stern 188 (e.g., rear oraft part) of watercraft 180.

Thus, in one implementation, a plurality of infrared capture components130A-130D (e.g., infrared cameras) may be mounted around the perimeterof watercraft 180 to provide fields of view thereabout. As an example,watercraft 180 may incorporate infrared imaging system 100B to provideman overboard detection, to assist during various modes of operation,such as night docking, night cruising, and/or day cruising of watercraft180, and/or to provide various information, such as improved imageclarity during hazy conditions or to provide a visual indication of thehorizon and/or shoreline.

FIG. 1D shows a top-view of infrared imaging system 100B having aplurality of image capture components 130E-130H (e.g., infrared cameras)mounted to a control tower 190 (e.g., bridge) of watercraft 180 inaccordance with an embodiment of the present disclosure. As shown inFIG. 1D, a plurality of image capture components 130E-130H may bemounted to control tower 190 in a configuration at different positionson watercraft 180 in a manner so as to provide one or more fields ofview around watercraft 180. In various implementations, image capturecomponent 130E may be mounted to provide a field of view of bow 182 ofwatercraft 180. As further shown, image capture component 130F may bemounted to provide a field of view of port 184 of watercraft 180. Asfurther shown, image capture component 130G may be mounted to provide afield of view of starboard 186 of watercraft 180. As further shown,image capture component 130H may be mounted to provide a field of viewof stern 188 of watercraft 180. Thus, in one implementation, a pluralityof image capture components 130E-130H (e.g., infrared cameras) may bemounted around control tower 190 of watercraft 180 to provide fields ofview thereabout. Furthermore as shown, image capture components 130B and130C may also be mounted on control tower 190 of watercraft 180.

FIG. 1E shows the port-side-view of infrared imaging system 100B havingport-side image capture component 130B of FIG. 1B mounted to watercraft180 in accordance with an embodiment of the present disclosure. Inreference to FIG. 1E, image capture component 130B provides a port-sidefield of view around watercraft 180.

In one implementation, image capture component 130B may provide a fieldof view of a port-side image of watercraft 180. In anotherimplementation, the port-side field of view may be segmented into aplurality of views B₁-B₆. For example, image capture component 130B maybe adapted to provide one or more segmented narrow fields of view of theport-side field of view including one or more forward port-side viewsB₁-B₃ and one or more rearward port-side views B₄-B₆. In still anotherimplementation, as shown in FIG. 6, image capture component 130B maycomprise a plurality of image capture components 132 (and optionally aplurality of image capture components 134) to provide the plurality ofsegmented or narrowed fields of view B₁-B₆ within the overall port-sidefield of view of watercraft 180.

As further shown in FIG. 1E, the port-side fields of view B₁-B₆ ofwatercraft 180 may extend through a viewing range from image capturecomponent 130B to a water surface 198 adjacent to watercraft 180.However, in various implementations, the viewing range may include aportion below the water surface 198 depending on the type of infrareddetector utilized (e.g., type of infrared camera, desired wavelength orportion of the infrared spectrum, and other relevant factors as would beunderstood by one skilled in the art).

FIG. 1F shows an example of locating and identifying a man overboardwithin the port-side field of view of port-side image capture component130B mounted to watercraft 180 in accordance with an embodiment of thepresent disclosure. In general, image capture component 130B may be usedto identify and locate a man overboard (e.g., within the narrowedport-side field of view B₃) of watercraft 180. Once the man overboard isidentified and located, processing component 110 of infrared imagingsystem 100B may control or provide information (e.g., slew-to-queue) toposition a searchlight component 136 of FIG. 6 within the port-sidefield of view B₃ to aid in visual identification and rescue of the manoverboard. It should be understood that searchlight component 136 may beseparate from image capture component 130B (e.g., separate housingand/or control) or may be formed as part of image capture component 130B(e.g., within the same housing or enclosure).

FIG. 2 shows a method 200 for capturing and processing infrared imagesin accordance with an embodiment of the present disclosure. For purposesof simplifying discussion of FIG. 2, reference may be made to imagecapturing systems 100A, 100B of FIGS. 1A, 1B as an example of a system,device or apparatus that may perform method 200.

Referring to FIG. 2, an image (e.g., infrared image signal) is captured(block 210) with infrared imaging system 100A, 100B. In oneimplementation, processing component 110 induces (e.g., causes) imagecapture component 130 to capture an image, such as, for example, image170. After receiving the captured image from image capture component130, processing component 110 may optionally store the captured image inmemory component 120 for processing.

Next, the captured image may optionally be pre-processed (block 215). Inone implementation, pre-processing may include obtaining infrared sensordata related to the captured image, applying correction terms, and/orapplying temporal noise reduction to improve image quality prior tofurther processing. In another implementation, processing component 110may directly pre-process the captured image or optionally retrieve thecaptured image stored in memory component 120 and then pre-process theimage. Pre-processed images may be optionally stored in memory component120 for further processing.

Next, a selected mode of operation may be obtained (block 220). In oneimplementation, the selected mode of operation may comprise a user inputcontrol signal that may be obtained or received from control component150 (e.g., control panel unit 500 of FIG. 5). In variousimplementations, the selected mode of operation may be selected from atleast one of night docking, man overboard, night cruising, day cruising,hazy conditions, and shoreline mode. As such, processing component 110may communicate with control component 150 to obtain the selected modeof operation as input by a user. These modes of operation are describedin greater detail herein and may include the use of one or more infraredimage processing algorithms.

In various implementations, modes of operation refer to presetprocessing and display functions for an infrared image, and infraredimagers and infrared cameras are adapted to process infrared sensor dataprior to displaying the data to a user. In general, display algorithmsattempt to present the scene (i.e., field of view) information in aneffective way to the user. In some cases, infrared image processingalgorithms are utilized to present a good image under a variety ofconditions, and the infrared image processing algorithms provide theuser with one or more options to tune parameters and run the camera in“manual mode”. In one aspect, infrared imaging system 100A, 100B may besimplified by hiding advanced manual settings. In another aspect, theconcept of preset image processing for different conditions may beimplemented in maritime applications.

Next, referring to FIG. 2, the image is processed in accordance with theselected mode of operation (block 225), in a manner as described ingreater detail herein. In one implementation, processing component 110may store the processed image in memory component 120 for displaying. Inanother implementation, processing component 110 may retrieve theprocessed image stored in memory component 120 and display the processedimage on display component 150 for viewing by a user.

Next, a determination is made as to whether to display the processedimage in a night mode (block 230). If yes, then processing component 110configures display component 140 to apply a night color palette to theprocessed image (block 235), and the processed image is displayed innight mode (block 240). For example, in night mode (e.g., for nightdocking, night cruising, or other modes when operating at night), animage may be displayed in a red palette or green palette to improvenight vision capacity for a user. Otherwise, if night mode is notnecessary, then the processed image is displayed in a non-night modemanner (e.g., black hot or white hot palette) (block 240).

In various implementations, the night mode of displaying images refersto using a red color palette or green color palette to assist the useror operator in the dark when adjusting to low light conditions. Duringnight operation of image capturing system 100A, 100B, human visualcapacity to see in the dark may be impaired by the blinding effect of abright image on a display monitor. Hence, the night mode setting changesthe color palette from a standard black hot or white hot palette to ared or green color palette display. In one aspect, the red or greencolor palette is generally known to interfere less with human nightvision capacity. In one example, for a red-green-blue (RGB) type ofdisplay, the green and blue pixels may be disabled to boost the redcolor for a red color palette. In another implementation, the night modedisplay may be combined with any other mode of operation of infraredimaging system 100A, 100B, as described herein, and a default displaymode of infrared imaging system 100A, 100B at night may be the nightmode display.

Furthermore in various implementations, certain image features may beappropriately marked (e.g., color-indicated or colorized, highlighted,or identified with other indicia), such as during the image processing(block 225) or displaying of the processed image (block 240), to aid auser to identify these features while viewing the displayed image. Forexample, as discussed further herein, during a man overboard mode, asuspected person (e.g., or other warm-bodied animal or object) may beindicated in the displayed image with a blue color (or other color ortype of marking) relative to the black and white palette or night colorpalette (e.g., red palette). As another example, as discussed furtherherein, during a nighttime or daytime cruising mode and/or hazyconditions mode, potential hazards in the water may be indicated in thedisplayed image with a yellow color (or other color or type of marking)to aid a user viewing the display. Further details regarding imagecolorization may be found, for example, in U.S. Pat. No. 6,849,849,which is assigned to Applicant, and which is thus incorporated herein byreference in its entirety.

In various implementations, processing component 110 may switch theprocessing mode of a captured image in real time and change thedisplayed processed image from one mode, corresponding to mode modules112A-112N, to a different mode upon receiving user input from controlcomponent 150. As such, processing component 110 may switch a currentmode of display to a different mode of display for viewing the processedimage by the user or operator on display component 140. This switchingmay be referred to as applying the infrared camera processing techniquesof mode modules 112A-112N for real time applications, wherein a user oroperator may change the displayed mode while viewing an image on displaycomponent 140 based on user input to control component 150.

FIGS. 3A-3E show block diagrams illustrating infrared processingtechniques in accordance with various embodiments of the presentdisclosure. As described herein, infrared imaging system 100A, 100B isadapted to switch between different modes of operation so as to improvethe infrared images and information provided to a user or operator.

FIG. 3A shows one embodiment of an infrared processing technique 300 asdescribed in reference to block 225 of FIG. 2. In one implementation,the infrared processing technique 300 comprises a night docking mode ofoperation for maritime applications. For example, during night docking,a watercraft or sea vessel is in the vicinity of a harbor, jetty ormarina, each of which having proximate structures including piers,buoys, other watercraft, other structures on land. A thermal infraredimager (e.g., infrared imaging system 100A, 100B) may be used as anavigational tool in finding a correct docking spot. The infraredimaging system 100A, 100B produces an infrared image that assists theuser or operator in docking the watercraft. There is a high likelihoodof hotspots in the image, such as dock lights, vents and running motors,which may have a minimal impact on how the scene is displayed.

Referring to FIG. 3A, the input image is histogram equalized and scaled(e.g., 0-511) to form a histogram equalized part (block 302). Next, theinput image is linearly scaled (e.g., 0-128) while saturating thehighest and lowest parts or proportions (e.g., 1%) to form a linearlyscaled part (block 304). Next, the histogram-equalized part and thelinearly scaled part are added together to form an output image (block306). Next, the dynamic range of the output image is linearly mapped tofit the display component 140 (block 308). It should be appreciated thatthe block order in which the process 300 is executed may be executed ina different order without departing from the scope of the presentdisclosure.

In one embodiment, the night docking mode is intended for image settingswith large amounts of thermal clutter, such as a harbor, a port, or ananchorage. The settings may allow the user to view the scene withoutblooming on hot objects. Hence, infrared processing technique 300 forthe night docking mode is useful for situational awareness in maritimeapplications when, for example, docking a watercraft with lowvisibility.

In various implementations, during processing of an image when the nightdocking mode is selected, the image is histogram equalized to compressthe dynamic range by removing “holes” in the histogram. The histogrammay be plateau limited so that large uniform areas, such as sky or watercomponents, are not given too much contrast. For example, approximately20% of the dynamic range of the output image may be preserved for astraight linear mapping of the non-histogram equalized image. In thelinear mapping, for example, the lowest 1% of the pixel values aremapped to zero and the highest 1% of the input pixels are mapped to amaximum value of the display range (e.g., 235). In one aspect, the finaloutput image becomes a weighted sum of the histogram equalized andlinearly (with 1% “outlier” cropping) mapped images.

FIG. 3B shows one embodiment of an infrared processing technique 320 asdescribed in reference to block 225 of FIG. 2. In one implementation,the infrared processing technique 320 comprises a man overboard mode ofoperation for maritime applications. For example, in the man overboardmode, image capturing system 100A, 100B may be tuned to the specifictask of finding a person in the water. The distance between the personin the water and the watercraft may not be known, and the person may beonly a few pixels in diameter or significantly larger if lying close tothe watercraft. In one aspect, even though a person may be close to thewatercraft, the person may have enough thermal signature to be clearlyvisible, and thus the man overboard display mode may target the casewhere the person has weak thermal contrast and is far enough away so asto not be clearly visible without the aid of image capturing system100A, 100B.

Referring to FIG. 3B, image capture component 130 (e.g., infraredcamera) of image capturing system 100A, 100B is positioned to resolve oridentify the horizon (block 322). In one implementation, the infraredcamera is positioned so that the horizon is at an upper part of thefield of view (FoV). In another implementation, the shoreline may alsobe indicated along with the horizon. Next, a high pass filter (HPF) isapplied to the image to form an output image (block 324). Next, thedynamic range of the output image is linearly mapped to fit the displaycomponent 140 (block 326). It should be appreciated that the block orderin which the process 320 is executed may be executed in a differentorder without departing from the scope of the present disclosure.

In one example, horizon identification may include shorelineidentification, and the horizon and/or shoreline may be indicated by aline (e.g., a red line or other indicia) superimposed on a thermal imagealong the horizon and/or the shoreline. Such indication may be usefulfor user or operators to determine position of the watercraft inrelation to the shoreline. Horizon and/or shoreline identification maybe accomplished by utilizing a real-time Hough transform or otherequivalent type of transform applied to the image stream, wherein thisimage processing transform finds linear regions (e.g., lines) in animage. The real-time Hough transform may also be used to find thehorizon and/or shoreline in open ocean when, for example, the contrastmay be low. Under clear conditions, the horizon and/or shoreline may beeasy identified. However, on a hazy day, the horizon and/or shorelinemay be difficult to locate.

In general, knowing where the horizon and/or shoreline are is useful forsituational awareness. As such, in various implementations, the Houghtransform may be allied to any of the modes of operation describedherein to identify the horizon and/or shoreline in an image. Forexample, the shoreline identification (e.g., horizon and/or shoreline)may be included along with any of the processing modes to provide a line(e.g., any type of marker, such as a red line or other indicia) on thedisplayed image and/or the information may be used to position theinfrared camera's field of view.

In one embodiment of the man overboard mode, signal gain may beincreased to bring out minute temperature differences of the ocean, suchas encountered when looking for a hypothermic body in a uniform oceantemperature that may be close to the person's body temperature. Imagequality is traded for the ability to detect small temperature changeswhen comparing a human body to ocean temperature. Thus, infraredprocessing technique 320 for the man overboard mode is useful forsituational awareness in maritime applications when, for example,searching for a man overboard proximate to the watercraft.

In various implementations, during processing of an image when the manoverboard mode is selected, a high pass filter is applied to the image.For example, the signal from the convolution of the image by a Gaussiankernel may be subtracted. The remaining high pass information islinearly stretched to fit the display range, which may increase thecontrast of any small object in the water. In one enhancement of the manoverboard mode, objects in the water may be marked, and the systemsignals the watercraft to direct a searchlight at the object. Forsystems with both visible and thermal imagers, the thermal imager isdisplayed. For zoom or multi-FoV systems, the system is set in a wideFoV. For pan-tilt controlled systems with stored elevation settings forthe horizon, the system is moved so that the horizon is visible justbelow the upper limit of the field of view.

In one embodiment, the man overboard mode may activate a locateprocedure to identify an area of interest, zoom-in on the area ofinterest, and position a searchlight on the area of interest. Forexample, the man overboard mode may activate a locate procedure toidentify a position of a object (e.g., a person) in the water, zoom-inthe infrared imaging device (e.g., an infrared camera) on the identifiedobject in the water, and then point a searchlight on the identifiedobject in the water. In another embodiment, the man overboard mode maybe adapted to maintain tracking of an area of interest as thesurveillance craft moves relative to the region of interest and/or theregion of interest drifts relative to the surveillance craft. In variousimplementations, these actions may be added to process 200 of FIG. 2and/or process 320 of FIG. 3B and further be adapted to occurautomatically so that the area of interest and/or location of the objectof interest may be quickly identified and retrieved by a crew member.

FIG. 3C shows one embodiment of an infrared processing technique 340 asdescribed in reference to block 225 of FIG. 2. In one implementation,the infrared processing technique 340 comprises a night cruising mode ofoperation for maritime applications. For example, during night cruising,the visible channel has limited use for other than artificiallyilluminated objects, such as other watercraft. The thermal infraredimager may be used to penetrate the darkness and assist in theidentification of buoys, rocks, other watercraft, islands and structureson shore. The thermal infrared imager may also find semi-submergedobstacles that potentially lie directly in the course of the watercraft.In the night cruising mode, the display algorithm may be tuned to findobjects in the water without distorting the scene (i.e., field of view)to the extent that it becomes useless for navigation.

In one embodiment, the night cruising mode is intended for low contrastsituations encountered on an open ocean. The scene (i.e., field of view)may be filled with a uniform temperature ocean, and any navigationalaids or floating debris may sharply contrast with the uniformtemperature of the ocean. Therefore, infrared processing technique 340for the night cruising mode is useful for situational awareness in, forexample, the open ocean.

Referring to FIG. 3C, the image is separated into a background imagepart and a detailed image part (block 342). Next, the background imagepart is histogram equalized (block 344) and scaled (e.g., 0-450) (block346). Next, the detailed image part is scaled (e.g., 0-511) (block 348).Next, the histogram-equalized background image part and the scaleddetailed image part are added together to form an output image (block350). Next, the dynamic range of the output image is linearly mapped tofit the display component 140 (block 352). It should be appreciated thatthe block order in which the process 340 is executed may be executed ina different order without departing from the scope of the presentdisclosure.

In various implementations, during processing of an image when the nightcruising mode is selected, the input image is split into detailed andbackground image components using a non-linear edge preserving low passfilter (LPF), such as a median filter or by anisotropic diffusion. Thebackground image component comprises a low pass component, and thedetailed image part is extracted by subtracting the background imagepart from the input image. To enhance the contrast of small andpotentially weak objects (e.g., with potentially weak thermalsignatures), the detailed and background image components may be scaledso that the details are given approximately 60% of the output/displaydynamic range. In one enhancement of the night cruising mode, objects inthe water are tracked, and if they are on direct collision course withthe current watercraft course, then they are marked in the image, and anaudible and/or visual alarm may be sounded and/or displayed,respectively. In some implementations, for systems with both visible andthermal imagers, the thermal imager may be displayed by default.

In one embodiment, a first part of the image signal may include abackground image part comprising a low spatial frequency high amplitudeportion of an image. In one example, a low pass filter (e.g., low passfilter algorithm) may be utilized to isolate the low spatial frequencyhigh amplitude portion of the image signal (e.g., infrared imagesignal). In another embodiment, a second part of the image signal mayinclude a detailed image part comprising a high spatial frequency lowamplitude portion of an image. In one example, a high pass filter (e.g.,high pass filter algorithm) may be utilized to isolate the high spatialfrequency low amplitude portion of the image signal (e.g., infraredimage signal). Alternately, the second part may be derived from theimage signal and the first part of the image signal, such as bysubtracting the first part from the image signal.

In general for example, the two image parts (e.g., first and secondparts) of the image signal may be separately scaled before merging thetwo image parts to produce an output image. For example, the first orsecond parts may be scaled or both the first and second parts may bescaled. In one aspect, this may allow the system to output an imagewhere fine details are visible and tunable even in a high dynamic rangescene. In some instances, as an example, if an image appears less usefulor degraded by some degree due to noise, then one of the parts of theimage, such as the detailed part, may be suppressed rather thanamplified to suppress the noise in the merged image to improve imagequality.

FIG. 3D shows one embodiment of an infrared processing technique 360 asdescribed in reference to block 225 of FIG. 2. In one implementation,the infrared processing technique 360 comprises a day cruising mode ofoperation for maritime applications. For example, during day cruising,the user or operator may rely on human vision for orientationimmediately around the watercraft. Image capturing system 100A, 100B maybe used to zoom in on objects of interest, which may involve reading thenames of other watercraft, and searching for buoys, structures on land,etc.

Referring to FIG. 3D, the image is separated into a background imagepart and a detailed image part (block 362). Next, the background imagepart is histogram equalized (block 364) and scaled (e.g., 0-511) (block366). Next, the detailed image part is scaled 0-255 (block 368). Next,the histogram-equalized background image part and the scaled detailedimage part are added together to form an output image (block 370). Next,the dynamic range of the output image is linearly mapped to fit thedisplay component 140 (block 372). It should be appreciated that theblock order in which the process 360 is executed may be executed in adifferent order without departing from the scope of the presentdisclosure.

In one embodiment, the day cruising mode is intended for higher contrastsituations, such as when solar heating leads to greater temperaturedifferences between unsubmerged or partially submerged objects and theocean temperature. Hence, infrared processing technique 360 for the daycruising mode is useful for situational awareness in, for example, highcontrast situations in maritime applications.

In various implementations, during processing of an image when the daycruising mode is selected, the input image is split into its detailedand background components respectively using a non-linear edgepreserving low pass filter, such as a median filter or by anisotropicdiffusion. For color images, this operation may be achieved on theintensity part of the image (e.g., Y in a YCrCb format). The backgroundimage part comprises the low pass component, and the detailed image partmay be extracted by subtracting the background image part from the inputimage. To enhance the contrast of small and potentially weak objects(e.g., with potentially weak thermal signatures), the detailed andbackground image parts may be scaled so that the details are givenapproximately 35% of the output/display dynamic range. For systems withboth visible and thermal imagers the visible image may be displayed bydefault.

FIG. 3E shows one embodiment of an infrared processing technique 380 asdescribed in reference to block 225 of FIG. 2. In one implementation,the infrared processing technique 380 comprises a hazy conditions modeof operation for maritime applications. For example, even during daytimeoperation, a user or operator may achieve better performance from animager using an infrared (MWIR, LWIR) or near infrared (NIR) wave band.Depending on vapor content and particle size, a thermal infrared imagermay significantly improve visibility under hazy conditions. If neitherthe visible nor the thermal imagers penetrate the haze, image capturingsystem 100A, 100B may be set in hazy conditions mode under which system100A, 100B attempts to extract what little information is available fromthe chosen infrared sensor. Under hazy conditions, there may be littlehigh spatial frequency information (e.g., mainly due, in one aspect, toscattering by particles). The information in the image may be obtainedfrom the low frequency part of the image, and boosting the higherfrequencies may drown the image in noise (e.g., temporal and/or fixedpattern).

Referring to FIG. 3E, a non-linear edge preserving low pass filter (LPF)is applied to the image (block 382). Next, the image is histogramequalized (block 384) and scaled (block 386) to form a histogramequalized output image. Next, the dynamic range of the output image islinearly mapped to fit the display component 140 (block 388). It shouldbe appreciated that the block order in which the process 380 is executedmay be executed in a different order without departing from the scope ofthe present disclosure.

In various implementations, during processing of an image when the hazyconditions mode is selected, a non-linear, edge preserving, low passfilter, such as median or by anisotropic diffusion is applied to theimage (i.e., either from the thermal imager or the intensity componentof the visible color image). In one aspect, the output from the low passfilter operation may be histogram equalized and scaled to map thedynamic range to the display and to maximize contrast of the display.

FIG. 3F shows one embodiment of an infrared processing technique 390 asdescribed in reference to block 225 of FIG. 2. In one implementation,the infrared processing technique 390 comprises a shoreline mode ofoperation for maritime applications.

Referring to FIG. 3F, the shoreline may be resolved (block 392). Forexample, shoreline identification (e.g., horizon and/or shoreline) maybe determined by applying an image processing transform (e.g., a Houghtransform) to the image (block 392), which may be used to position theinfrared camera's field of view and/or to provide a line (e.g., any typeof marker, such as a red line(s) or other indicia on the displayedimage. Next, the image is histogram equalized (block 394) and scaled(block 396) to form an output image. Next, the dynamic range of theoutput image is linearly mapped to fit the display component 140 (block398). It should be appreciated that the block order in which the process390 is executed may be executed in a different order without departingfrom the scope of the present disclosure.

In one implementation, the information produced by the transform (e.g.,Hough transform) may be used to identify the shoreline or even thehorizon as a linear region for display. The transform may be applied tothe image in a path separate from the main video path (e.g., thetransform when applied does not alter the image data and does not affectthe later image processing operations), and the application of thetransform may be used to detect linear regions, such as straight lines(e.g., of the shoreline and/or horizon). In one aspect, by assuming theshoreline and/or horizon comprises a straight line stretching the entirewidth of the frame, the shoreline and/or horizon may be identified as apeak in the transform and may be used to maintain the field of view in aposition with reference to the shoreline and/or horizon. As such, theinput image (e.g., preprocessed image) may be histogram equalized (block394) and scaled (block 396) to generate an output image, and then thetransform information (block 392) may be added to the output image tohighlight the shoreline and/or horizon of the displayed image.

Moreover, in the shoreline mode of operation, the image may be dominatedby sea (i.e., lower part of image) and sky (i.e., upper part of image),which may appear as two peaks in the image histogram. In one aspect,significant contrast is desired over the narrow band of shoreline, and alow number (e.g., relatively based on the number of sensor pixels andthe number of bins used in the histogram) may be selected for theplateau limit for the histogram equalization. In one aspect, forexample, a low plateau limit (relative) may reduce the effect of peaksin the histogram and give less contrast to sea and sky while preservingcontrast for the shoreline and/or horizon regions.

FIG. 4 shows a block diagram illustrating a method 400 of implementingmodes 410A-410E and infrared processing techniques related thereto, asdescribed in reference to various embodiments of the present disclosure.In particular, a first mode refers to night docking mode 410A, a secondmode refers to man overboard mode 410B, a third mode refers to nightcruising mode 410C, a fourth mode refers to day cruising mode 410D, anda fifth mode refers to hazy conditions mode 410E.

In one implementation, referring to FIG. 4, processing component 110 ofimage capturing system 100A, 100B of FIGS. 1A, 1B may perform method 400as follows. Sensor data (i.e., infrared image data) of a captured imageis received or obtained (block 402). Correction terms are applied to thereceived sensor data (block 404), and temporal noise reduction isapplied to the received sensor data (block 406).

Next, at least one of the selected modes 410A-410E may be selected by auser or operator via control component 150 of image capturing system100A, 100B, and processing component 110 executes the correspondingprocessing technique associated with the selected mode of operation. Inone example, if night docking mode 410A is selected, then the sensordata may be histogram equalized and scaled (e.g., 0-511) (block 420),the sensor data may be linearly scaled (e.g., 0-128) saturating thehighest and lowest (e.g., 1%) (block 422), and the histogram equalizedsensor data is added to the linearly scaled sensor data for linearlymapping the dynamic range to display component 140 (block 424). Inanother example, if man overboard mode 410B is selected, then infraredcapturing component 130 of image capturing system 100A, 100B may bemoved or positioned so that the horizon is at an upper part of the fieldof view (FoV), a high pass filter (HPF) is applied to the sensor data(block 432), and the dynamic range of the high pass filtered sensor datais then linearly mapped to fit display component 140 (block 434). Inanother example, if night cruising mode 410C is selected, the sensordata is processed to extract a faint detailed part and a background partwith a high pass filter (block 440), the background part is histogramequalized and scaled (e.g., 0-450) (block 442), the detailed part isscaled (e.g., 0-511) (block 444), and the background part is added tothe detailed part for linearly mapping the dynamic range to displaycomponent 140 (block 446). In another example, if day cruising mode 410Dis selected, the sensor data is processed to extract a faint detailedpart and a background part with a high pass filter (block 450), thebackground part is histogram equalized and scaled (e.g., 0-511) (block452), the detailed part is scaled 0-255 (block 454), and the backgroundpart is added to the detailed part for linearly mapping the dynamicrange to display component 140 (block 456). In still another example, ifhazy condition mode 410E is selected, then a non-linear low pass filter(e.g., median) is applied to the sensor data (block 460), which is thenhistogram equalized and scaled for linearly mapping the dynamic range todisplay component 140 (block 462).

For any of the modes (e.g., blocks 410A-410E), the image data fordisplay may be marked (e.g., color coded, highlighted, or otherwiseidentified with indicia) to identify, for example, a suspected person inthe water (e.g., for man overboard) or a hazard in the water (e.g., fornighttime cruising, daytime cruising, or any of the other modes). Forexample, image processing algorithms may be applied (block 470) to theimage data to identify various features (e.g., objects, such as awarm-bodied person, water hazard, horizon, or shoreline) in the imagedata and appropriately mark these features to assist in recognition andidentification by a user viewing the display. As a specific example, asuspected person in the water may be colored blue, while a water hazard(e.g., floating debris) may be colored yellow in the displayed image.

Furthermore for any of the modes (e.g., blocks 410A-410E), the imagedata for display may be marked to identify, for example, the shoreline(e.g., shoreline and/or horizon). For example, image processingalgorithms may be applied (block 475) to the image data to identify theshoreline and/or horizon and appropriately mark these features to assistin recognition and identification by a user viewing the display. As aspecific example, the horizon and/or shoreline may be outlined oridentified with red lines on the displayed image to aid the user viewingthe displayed image.

Next, after applying at least one of the infrared processing techniquesfor modes 410A-410E, a determination is made as to whether to displaythe processed sensor data in night mode (i.e., apply the night colorpalette) (block 480). If yes, then the night color palette is applied tothe processed sensor data (block 482), and the processed sensor data isdisplayed in night mode (block 484). If no, then the processed sensordata is displayed in a non-night mode manner (e.g., black hot or whitehot palette) (block 484). It should be appreciated that, in night mode,sensor data (i.e., image data) may be displayed in a red or green colorpalette to improve night vision capacity for a user or operator.

FIG. 5 shows a block diagram illustrating one embodiment of controlcomponent 150 of infrared imaging system 100A, 100B for selectingbetween different modes of operation, as previously described inreference to FIGS. 2-4. In one embodiment, control component 150 ofinfrared imaging system 100A, 100B may comprise a user input and/orinterface device, such as control panel unit 500 (e.g., a wired orwireless handheld control unit) having one or more push buttons 510,520, 530, 540, 550, 560, 570 adapted to interface with a user andreceive user input control values and further adapted to generate andtransmit one or more input control signals to processing component 100A,100B. In various other embodiments, control panel unit 500 may comprisea slide bar, rotatable knob to select the desired mode, keyboard, etc.,without departing from the scope of the present disclosure.

In various implementations, a plurality of push buttons 510, 520, 530,540, 550, 560, 570 of control panel unit 500 may be utilized to selectbetween various modes of operation as previously described in referenceto FIGS. 2-4. In various implementations, processing component 110 maybe adapted to sense control input signals from control panel unit 500and respond to any sensed control input signals received from pushbuttons 510, 520, 530, 540, 550, 560, 570. Processing component 110 maybe further adapted to interpret the control input signals as values. Invarious other implementations, it should be appreciated that controlpanel unit 500 may be adapted to include one or more other push buttons(not shown) to provide various other control functions of infraredimaging system 100A, 100B, such as auto-focus, menu enable andselection, field of view (FoV), brightness, contrast, and/or variousother features. In another embodiment, control panel unit 500 maycomprise a single push button, which may be used to select each of themodes of operation 510, 520, 530, 540, 550, 560, 570.

In another embodiment, control panel unit 500 may be adapted to beintegrated as part of display component 140 to function as both a userinput device and a display device, such as, for example, a useractivated touch screen device adapted to receive input signals from auser touching different parts of the display screen. As such, the GUIinterface device may have one or more images of, for example, pushbuttons 510, 520, 530, 540, 550, 560, 570 adapted to interface with auser and receive user input control values via the touch screen ofdisplay component 140.

In one embodiment, referring to FIG. 5, a first push button 510 may beenabled to select the night docking mode of operation, a second pushbutton 520 may be enabled to select the man overboard mode of operation,a third push button 530 may be enabled to select the night cruising modeof operation, a fourth push button 540 may be enabled to select the daycruising mode of operation, a fifth push button 550 may be enabled toselect the hazy conditions mode of operation, a sixth push button 560may be enabled to select the shoreline mode of operation, and a seventhpush button 570 may be enabled to select or turn the night display mode(i.e., night color palette) off. In another embodiment, a single pushbutton for control panel unit 500 may be used to toggle t each of themodes of operation 510, 520, 530, 540, 550, 560, 570 without departingfrom the scope of the present disclosure.

FIG. 6 shows a schematic diagram illustrating an embodiment of imagecapture component 130 of infrared imaging system 100A, 100B. As shown,image capture component 130 may be adapted to comprise a first cameracomponent 132, a second camera component 134, and/or a searchlightcomponent 136. In various implementations, each of the components 132,134, 136 may be integrated as part of image capture component 130 or oneor more of the components 132, 134, 136 may be separate from imagecapture component 130 without departing from the scope of the presentdisclosure.

In one embodiment, first camera component 132 may comprise an infraredcamera component capable of capturing infrared image data of image 170.In general, an infrared camera is a device that is adapted to form animage using infrared radiation, which may be useful for rescueoperations in water and/or darkness.

In one embodiment, second camera component 134 may comprise anotherinfrared camera component or a camera capable of capturing visiblespectrum images of image 170. In general, a visible-wavelength cameramay be used by a crew member of watercraft 180 to view and examine theimage 170. For example, in daylight, the visible-wavelength camera mayassist with viewing, identifying, and locating a man overboard.

In various implementations, the camera components 132, 134 may beadapted to include a wide and/or narrow field of view (e.g., a fixed orvariable field of view). For example, this feature may include atelescoping lens that narrows the field of view to focus on a particulararea within the field of view.

In one embodiment, searchlight component 136 comprises a device capableof projecting a beam of light towards image 170 in the field of view. Inone implementation, searchlight component 136 is adapted to focus a beamof light on a target within the field of view of at least one of cameracomponents 132, 134 so as to identify and locate, for example, aposition of a man overboard, which would allow a crew member ofwatercraft 180 to have improved visibility of the man overboard indarkness.

FIG. 7 shows a block diagram illustrating an embodiment of a method 700for monitoring image data of infrared imaging system 100A, 100B. In oneimplementation, method 700 is performed by processing component 110 ofinfrared imaging system 100A, 100B. As shown in FIG. 7, image data isobtained (block 710). In various implementations, the image data may beobtained directly from the image capture component 130 or from storagein memory component 120.

Next, the obtained image data may be processed (block 714). In oneimplementation, the obtained image data may be processed using the manoverboard mode of operation 320 of FIG. 3B to collect image data todetect an object, such as a person, falling into or in the waterproximate to watercraft 180.

Next, a man overboard (e.g., person) may be identified from theprocessed image data (block 718). In one implementation, the object(e.g., a person) may be separated from the water based on thetemperature difference therebetween. For example, when a person having abody temperature of approximately 98 degrees falls into the water havinga water temperature of approximately 60-70 degrees or less, thedifference between the temperatures is viewable with an infrared image,and therefore, the person may be quickly identified and located in thewater.

In an example embodiment, various types of conventional image processingsoftware (e.g., a software package by ObjectVideo located in Reston,Va.) may be run by processing component 110 to perform image analysis tomonitor the image data and detect a man overboard condition. In anexample embodiment, features in such conventional software may supportthe use of threshold conditions or object discrimination, for example,to distinguish non-living objects, such as a deck chair or otherinanimate objects, from a person. Programming the software package withthreshold factors such as temperature, shape, size, aspect ratio,velocity, or other factors may assist a software package indiscriminating images of non-living and/or non-human objects from imagesof humans. Thus, threshold conditions for use as desired in a givenapplication may provide that a bird flying through a camera's field ofview, for example, may be ignored, as would a falling deck chair or cupof hot coffee thrown overboard.

When a man overboard condition is suspected or determined, an operator(e.g., crew member) may be alerted or notified (block 722) so that arescue action may be initiated. In various implementations, this alertor notification may comprise an audio signal and/or visual signal, suchas an alarm, a warning light, a siren, a bell, a buzzer, etc.

Next, the specific location of the man overboard may be identified basedon the image data (block 726). In one implementation, identifying thelocation of the person may include narrowing the field of view of theimage capture component 130. For example, a lens of the infrared cameramay telescope to a position to zoom-in on the object or person in thewater or zoom-in on at least the proximate location of the person in thewater or another narrower field of view image capture component 130 maybe directed to the proximate location of the person in the water.Furthermore, a searchlight (e.g., searchlight component 136 of the imagecapture component 130) may be directed to the proximate location of theperson in the water (block 730) to assist with the retrieval and rescueof the person overboard.

When a man overboard condition is detected, for example in accordancewith an embodiment, the time and/or location of the event may berecorded along with the image data (e.g., as part of block 722 or 726),so as to aid in the search and rescue operation and/or to provideinformation for later analysis of the suspected man overboard event.Alternatively, the time and/or location may be regularly recorded withthe image data. For example, processing component 110 (FIGS. 1 a, 1 b)may include a location determination function (e.g., a globalpositioning system (UPS) receiver or by other conventional locationdetermination techniques) to receive precise location and/or timeinformation, which may be stored (e.g., in memory component 120) alongwith the image data. The image data along with the location informationand/or time information may then be used, for example, to allow a searchand rescue crew to leave the ship (e.g., cruise ship) and backtrack in asmaller vessel or helicopter to the exact location of the man overboardcondition in a prompt fashion as a large ship generally would not beable to quickly stop and return to the location of the man overboardevent.

In accordance with embodiments of the present disclosure, operators mayutilize an enhanced vision system (EVS) to pilot and/or navigatevehicles (e.g., land-based vehicles including automobiles, air-basedvehicles including aircraft, and water-based vehicles includingwatercraft) in varying environmental conditions. For example, an EVS mayutilize multiple sensors including at least one sensor sensitive tovisible light (e.g., an optical sensor) to provide viewable images inambient light and at least one sensor sensitive to infrared radiation(e.g., IR sensors) to provide viewable images in darkness. Incircumstances where visible light sources exist (e.g., airport lights,tower lights, buoy lights, street lights, etc.), it may be desirable forthe operator to see those visible light sources, wherein a sensorsensitive to the visible light spectrum is able to image visible lightsources. In circumstances where visible light sources exist but haverelatively invisible structural features when viewed in darkness, it maybe desirable for the operator to see those structural features, whereina sensor sensitive to the infrared spectrum is capable of imagingrelatively invisible heat sources of structural features. In accordancewith one or more embodiments of the present disclosure, the systems100A, 100B of FIGS. 1A, 1B may be adapted to have enhanced visioncapability for combining images of visible and infrared wavelengthsensors. In one aspect, combining or blending image signals from dualsensors (e.g., combining image signals from a visible sensor with imagesignals from an infrared sensor) may be utilized to generate combined orblended image signals that retain, e.g., color information from thevisible sensor and shows, e.g., infrared luminance (e.g., irradiation)from the infrared sensor. In another aspect, three different types ofimage display modes may be provided by such a system, such as a visibleonly mode, an infrared only mode, and a combined or blended mode.

FIG. 8 shows one embodiment of a block diagram of a method 800 forimplementing dual sensor applications in an enhanced vision system. Asshown in FIG. 8, a plurality of signal paths (e.g., video signal paths)may be utilized in an enhanced vision system, such as system 100A ofFIG. 1A and/or system 100B of FIG. 1B adapted to have enhanced visioncapability. The enhanced vision systems 100A, 100B may be adapted tooperate in at least one of three modes, such as a first mode (i.e.,mode 1) with color or monochrome imagery from a visible light sensor, asecond mode (i.e., mode 2) with monochrome or pseudo color imagery froman infrared sensor, and a third mode (i.e., mode 3) with color ormonochrome imagery created by blending or combining image signals fromthe visible light sensor and the infrared sensor. It should beappreciated that the various modes may be described below in referenceto digital video signals; however, similar systems may use analog videosignals without departing from the scope of the present disclosure.

In one implementation, the image capture component 130 of the system100A of FIG. 1A may comprise one or more visible light sensors forcapturing visible image signals representative of the image 170 and oneor more infrared sensors for capturing infrared image signalsrepresentative of the image 170. Similarly, in another implementation,each of the image capture components 130A, 130B, 130C of the system 100Bof FIG. 1B may comprise one or more visible light sensors for capturingvisible image signals representative of the image 170 and one or moreinfrared sensors for capturing infrared image signals representative ofthe image 170.

Accordingly, in reference to FIG. 8, the image capture components 130,130A, 130B, 130C of the systems 100A, 100B, respectively, may compriseat least two sensors, wherein a first sensor is adapted to be sensitiveto ambient light in the visible spectrum to provide a visible spectrumimage signal 810 and a second sensor is adapted to be sensitive toinfrared radiation (i.e., thermal radiation) in the infrared spectrum toprovide an infrared spectrum image signal 830. In one aspect, the firstand second sensors may be arranged to have identical or at least partlyoverlapping fields of view (FOV). In another aspect, for improvedperformance, the first and second sensors may have synchronized framecapture capability such that image signals 810, 830 captured byrespective sensors are representative of the same image (e.g., the image170 or a scene representative of the image 170) as it appears at aboutthe same instant in time for a given pair of frames from the twosensors.

In mode 1, the visible spectrum image signal 810 (i.e., color image)from a visible light sensor is converted to the generally known YCrCbformat 814 (or any other format that separates luminance fromchrominance). Depending on the type of sensor and the condition underwhich it is used, some correction terms may be applied 812 to thevisible spectrum image signal 810. These correction terms may include,for example, lookup tables used to perform color correction on the image170. In mode 1, a user input control parameter 820 (e.g., implemented asat least one image control knob) is selected 824 and is adapted toaffect a luminance part (Y) 850 of only the visible image 810. Forexample, the user might control the brightness by applying luminancescaling 816 of the visible image 810 with the parameter ξ821. Afteroptional user control of the luminance signal by not selecting a blendedmode 826B, the potentially modified luminance part (Y) 850 is mergedwith the chrominance parts (Cr and Cb) 852 to form a color video stream860 for display. Optionally, the chrominance part (Cr and Cb) 852 may bediscarded to produce a monochrome video stream.

In mode 2, the infrared spectrum image signal 830 (i.e., infrared image)from an infrared sensor is received and correction terms (such as gainand offset correction) may be applied 832 to the infrared spectrum imagesignal 830. In one aspect, since infrared sensors tend to produce highdynamic range signals (e.g., up to 14 bit=16384 grey levels), dynamicrange compression and scaling may be performed 834 to adapt the infraredimage spectrum signal 830 for the display dynamic range (e.g., 8 bit=256levels of grey). This may be achieved by linear compression; however,non-linear methods, such as histogram equalization, may be utilized. Forexample, the user may control the dynamic range compression and scalingwith the single parameter ξ821 when an infrared only mode is selected822A with user input control parameter 820. After optional user controlof infrared luminance, a geometric transform may be applied 836 to thesignal 830. Next, if a blended mode is not selected 826C, the luminancepart (Y) 854 of the signal 830 may be directly sent as monochrome videoor pseudo colored using some predefined color palette to form aninfrared video stream for display 862.

In mode 3, the visible and infrared spectrum image signals 810, 830 areprocessed in a blended mode 826B, 826C using user control setting 820after not selecting the infrared only mode 822A and not selecting thevisible only mode 824. In mode 3, the user input control parameter 821is adapted to affect the proportions of the two luminance components850, 854 of the signals 810, 830, respectively. In one aspect, ξ821 maybe normalized with a value in the range of 0 (zero) to 1, wherein avalue of 1 produces a blended image 864 that is similar to the visibleimage 860 produced in mode 1. On the other hand, if ξ821 is set to 0,the blended image 864 is displayed with the luminance similar to theluminance of the infrared image 862. However, in the latter instance,the chrominance (Cr and Cb) 852 from the visible image 810 may beretained. Each other value of ξ821 is adapted to produce a blended image864 where the luminance part (Y) 850, 854, respectively, includesinformation from both the visible and infrared signals 810, 830,respectively. For example, after selecting the blended mode 826A, 821 ismultiplied 842 to the luminance part (Y) 850 of the visible image andadded 846 to the value obtained by multiplying 844 the value of 1−ξ 821to the luminance part (Y) 854 of the infrared image. This added valuefor the blended luminance parts (Y) 856 is used to provide the blendedimage 864.

It should be appreciated that the condition for infrared only modes822A, 822B may refer to the same condition being checked and/orverified, wherein an infrared only mode of operation is selected via theuser input control parameter 820. Moreover, it should also beappreciated that the condition for blended modes 826A, 826B, 826C refersto the same condition being checked and/or verified, wherein a blendedmode of operation is selected via the user input control parameter 820.

It should also be appreciated that, in one aspect, the enhanced visionsystem may automatically select the mode of operation based on the timeof day (e.g., day time or night time). It should also be appreciatedthat, in another aspect, the enhanced vision system may automaticallyselect the mode of operation based on properties of the captured imagesignals (e.g., SNR: signal-to-noise ratio).

In one embodiment, a blending algorithm may be referred to as true colorIR imagery. For example, in daytime imaging, a blended image maycomprise a visible color image, which includes a luminance element and achrominance element, with its luminance value replaced by the luminancevalue from the infrared image. The use of the luminance data from theinfrared image causes the intensity of the true visible color image tobrighten or dim based on the temperature of the object. As such, theblending algorithm provides IR imaging for daytime or visible lightimages.

Generally, luminance refers to a photometric measure of luminousintensity per unit area of light travelling in a given direction.Luminance characterizes emission or reflection of light from flat,diffuse surfaces. Luminance is an indicator of surface brightness as itmay appear to a human eye. In one aspect, luminance is used in the videoindustry to characterize the brightness of displays. Chrominance (i.e.,chroma) refers to a difference between a color and a reference color ofa same luminance (hue and saturation), wherein Y (luminance), Cr (reddiff chroma), and Cb (blue diff chroma) are utilized as image parts.Chrominance (CrCb) is an image signal that conveys color information ofan image separately from the accompanying luminance (Y) signal.Chrominance may be represented as two color-difference components: B′-Y′(blue-luma) and R′-Y′ (red-luma). Separating KGB (i.e., Red, Green,Blue) color signals into luminance and chrominance allows determinationof each color bandwidth to be calculated separately. In reference tovideo signals, luminance represents the brightness of an image and theachromatic image without any color, and the chrominance of an imagerepresents the color information.

FIG. 9 shows a block diagram illustrating a method 900 of implementingthe enhanced vision modes 1-3 of FIG. 8 and infrared processingtechniques related thereto, as described in reference to variousembodiments of the present disclosure. In one embodiment, mode 1 refersto a visible light processing mode of operation beginning with obtainingvisible sensor data 920 and preprocessing the visible sensor data 922including color video data, mode 2 refers to a thermal (infrared)processing mode of operation beginning with obtaining thermal sensordata 910 and preprocessing the thermal sensor data 912 including thermalvideo data, and mode 3 refers to a blending processing mode of operationadapted to blend the thermal sensor data 910 and the visible sensor data920 to produce a blended image signal including a blended video imagesignal.

In one embodiment, referring to FIG. 9, the thermal processing mode ofmethod 900 integrates modes 410A-410E of FIG. 4 and infrared processingtechniques related thereto, as described in reference to variousembodiments of the present disclosure. In particular, the thermal modesof operation include processing techniques related to the night dockingmode 410A, the man overboard mode 410B, the night cruising mode 410C,the day cruising mode 410D, and the hazy conditions mode 410E.

Referring to FIG. 9, the processing component 110 of the image capturingsystem 100A, 100B of FIGS. 1A, 1B is adapted to perform method 900 asfollows. Thermal sensor data (i.e., infrared image data) and visiblesensor data (i.e., color image data) of a captured image (e.g., image170) is received or obtained (blocks 910, 920, respectively). Next, thethermal sensor data 910 and the visible sensor data 920 are preprocessed(blocks 912, 922, respectively). In one aspect, preprocessing mayinclude applying correction terms and/or temporal noise reduction to thereceived sensor data.

In one implementation, after obtaining the thermal sensor data 910 andpreprocessing the thermal sensor data 912, at least one of the selectedmodes 410A-410E may be selected by a user or operator via controlcomponent 150 of image capturing system 100A, 100B, and processingcomponent 110 executes the corresponding processing technique associatedwith the selected mode. In one example, if night docking mode 410A isselected, then the sensor data may be histogram equalized and scaled(e.g., 0-511) (block 420), the sensor data may be linearly scaled (e.g.,0-128) saturating the highest and lowest (e.g., 1%) (block 422), and thehistogram equalized sensor data is added to the linearly scaled sensordata for linearly mapping the dynamic range to display component 140(block 424). In another example, if man overboard mode 410B is selected,then infrared capturing component 130 of image capturing system 100A,100B may be moved or positioned so that the horizon is at an upper partof the field of view (FoV), a high pass filter (HPF) is applied to thesensor data (block 432), and the dynamic range of the high pass filteredsensor data is then linearly mapped to fit display component 140 (block434). In another example, if night cruising mode 410C is selected, thesensor data is processed to extract a faint detailed part and abackground part with a high pass filter (block 440), the background partis histogram equalized and scaled (e.g., 0-450) (block 442), thedetailed part is scaled (e.g., 0-511) (block 444), and the backgroundpart is added to the detailed part for linearly mapping the dynamicrange to display component 140 (block 446). In another example, if daycruising mode 410D is selected, the sensor data is processed to extracta faint detailed part and a background part with a high pass filter(block 450), the background part is histogram equalized and scaled(e.g., 0-511) (block 452), the detailed part is scaled 0-255 (block454), and the background part is added to the detailed part for linearlymapping the dynamic range to display component 140 (block 456). In stillanother example, if hazy condition mode 410E is selected, then anon-linear low pass filter (e.g., median) is applied to the sensor data(block 460), which is then histogram equalized and scaled for linearlymapping the dynamic range to display component 140 (block 462).

In another implementation, after obtaining the visible sensor data 920and preprocessing the visible sensor data 922, the processing component110 is adapted to separate a luminance part (Y) and color part (CrCb)from the visible sensor data 920 (block 924). Next, if the blended modeis selected 950 by a user, the processing component 110 is adapted toblend the luminance part (Y) from the visible sensor data 920 with thescaled thermal data (block 926), as provided by at least one of themodes 410A-410E. Next, the processing component 110 is adapted to addchrominance values (CrCb) from the visible sensor data 920 (block 928)and display the blended mode image (block 952).

Otherwise, in one implementation, if the blended mode is not selected950 by the user, a determination is made as to whether to display theprocessed thermal sensor data in night mode (i.e., apply the night colorpalette) (block 480), in a manner as previously described. If yes, thenthe night color palette is applied to the processed sensor data (block482), and the processed sensor data is displayed in night mode (block484A). If no, then the processed thermal sensor data is displayed in anon-night mode manner (e.g., black hot or white hot palette) (block484B). It should be appreciated that, in night mode, thermal sensor data(i.e., infrared image data) may be displayed in a red or green colorpalette to improve night vision capacity for a user or operator.

Under certain conditions, it may be desirable for a vehicle (e.g.,aircraft, watercraft, land-based vehicle, etc.) to utilize an enhancedvision systems (EVS) to assist a pilot in operating and/or navigatingthe vehicle. For example, at nighttime, the use of a thermal sensorsensitive to infrared radiation may be advantageous since thermalsensors image even in darkness. In another example, under circumstanceswhere visible light sources exist, it may be desirable for the pilot tosee those lights. As such, a visible light sensor sensitive to thevisible light spectrum provides an ability to image visible lightsources. Accordingly, the methods 800, 900 of FIGS. 8-9, respectively,allow for combing images from infrared and visible wavelength sensorsfor vehicle applications. In one aspect, by combing image signals (e.g.,video signals) from a visible light sensor with image signals (e.g.,video signals) from an infrared sensor, a blended or combined imagesignal may be created that retains color information (if any) from thevisible light sensor and shows infrared luminance (irradiation) from theinfrared sensor. It should be appreciated that the methods 800, 900 maybe operated in a visible only mode, an infrared only mode, and avariable blended visible/infrared mode, without departing from the scopeof the present disclosure.

FIG. 10 shows a block diagram illustrating one embodiment of the controlcomponent 150 of the enhanced vision systems 100A, 100B for selectingbetween different modes of operation, as previously described inreference to FIGS. 2-4 and 8-9. In one embodiment, the control component150 of infrared imaging system 100A, 100B may comprise a user inputand/or interface device, such as control panel unit 1000 (e.g., a wiredor wireless handheld control unit) having one or more control knobs1080, 1090 and push buttons 510, 520, 530, 540, 550, 560, 570 (asdescribed in reference to FIG. 5) adapted to interface with a user andreceive user input control values and further adapted to generate andtransmit one or more input control signals to processing component 100A,100B.

In various other embodiments, the control panel unit 1000 may comprise aslide bar, push button, rotatable knob to select the desired mode,keyboard, etc., without departing from the scope of the presentdisclosure. For example, the control panel unit 1000 for the enhancedvision systems 100A, 100B comprises a selectable knob with fixedpositions, such as nine fixed positions, with hard stops at position 1and position 9. In one implementation, a first position (1) may beadapted to select an infrared only mode of operation, a ninth position(9) may be adapted to select a visible only mode of operation, andsecond to eighth intermediate positions (2 thru 8) may be adapted toselect and produce a blended video image having increasing amounts ofvisible luminance being added for each selected increment, which isdescribed in greater detail herein.

In another embodiment, the control panel unit 1000 may comprise ajoystick control device, such as a joystick control unit (JCU), anauto-centering JCU, etc., where proportional control is used to add(e.g., with a clockwise (CW) rotation) or subtract (e.g., with a counterclockwise (CCW) rotation) visible-band content from a starting admixtureof 50% (e.g., by default) or a user defined default admixture. At eitherrotational limiting position (e.g., CW or CCW), there is a hard stopthat represents 100% visible image spectrum (VIS) or 100% infrared imagespectrum (IIS) output, respectively. In one implementation, an on-screendisplay (OSD) overlay may be utilized to indicate a percentage (%) ofadmixture for blending luminance of visible and infrared spectrums.

In one embodiment, visible cameras and thermal cameras may only allowfor some maximum gain of the video signal. A gain limit ensures thatnoise components in the image are limited so that the video signal isusable. The gain limit may limit the visible or thermal video signalsuch that they do not use the full output dynamic range available. Forexample, the visible signal luminance component may only be 100 countsout of an available 256 counts in the 8-bit domain. In this case, it isadvantageous to rescale the two luminance components (i.e., visible andthermal luminance components) so that the visible part is relativelyless than what the user or system has selected. In one aspect, a usermay have set the blending factor such that half of the luminance (e.g.,ξ=0.5) in the blended output comes from the visible luminance channel.The remaining 50% of the luminance comes from the thermal videocomponent. In reference to the limited dynamic range of the visiblevideo component, an adjusted blending factor ξ_(a) may be calculatedbased on the control parameter (ξ) that the user may have selected suchthat ξ_(a)=(1−ξ)+ξ((256−100)/256)≈0.8, in one example. The luminanceblending algorithm may then be L_(blend)=ξ_(a)L_(ir)+ξL_(vis), whereL_(ir) and L_(vis) refer to the thermal and visible luminancecomponents, respectively, and L_(blend) is the resulting blendedluminance.

In one implementation, if the luminance is substantially low, e.g.,close to black, it may be difficult for a human observer to notice colorvariations. Hence, as an improvement of the luminance blending, thechrominance values may be utilized to adjust the blending percentages ona per pixel basis. In a YCrCb format, gray is represented by the Cr andCb values being at the midpoint of their dynamic range, e.g., 128 for8-bit representation. In this example, a pixel may be defined to containmore color information when deviating by varying degrees from a centervalue of its Cr and Cb channels. In one aspect, a measure of colorcontent may comprise the sum of absolute differences from gray, e.g.,C=abs(Cr−128)+abs(Cb−128), where C may be primitively interpreted asamount of color. Assuming the color is visible in the original visiblevideo stream, the blending factor (ξ) may be adjusted on a per pixellevel such that pixels with significant color content (e.g., largevalues for C) are weighted more towards the visible sensor luminance.For example, a blending parameter ξ_(i,j)(C_(i,j)) may be defined suchthat λ is monotonically increasing for increasing values of C, e.g., aGaussian function may be used, as shown in FIG. 11A. The color adjustedblending factor ξ_(c) may be defined as the sum of the user selectedblending factor ξ_(u) and the color factor λ. If ξ_(c) is limited to therange 0-1, then the blending equation may be writtenL_(blend)=(1−ξ_(c))L_(ir)+ξ_(c)L_(vis). As will be clear to someoneskilled in the art, other normalized weighting schemes may also be usedto shift luminance content towards the visible component for pixels withsignificant color content.

In another implementation, another approach to prevent color informationin dark areas to be lost in the blended video is to brighten dark areas,if they have color content. For example, this may be achieved by addinga luminance offset to all pixels with some color content (e.g., asdefined by the parameter C) above some predefined threshold. A smoothertransition is achieved if the offset is proportional to C. For largerabsolute differences from gray (e.g., large C) a larger luminance offsetis added. In one aspect, parts of the image that are already bright neednot be brightened further, and therefore, the brightness offset may belower for already bright areas. A brightness correction functionβ_(i,j)(C_(i,j),L) may be defined, where β is increasing for increasingvalues for C and decreasing for increasing values of L. An example ofsuch a function is shown in FIG. 11B.

In various implementations, a plurality of control knobs 1080, 1090 ofcontrol panel unit 1000 may be utilized to select between various modesof operation (i.e., visible display only, infrared display only, andblended display modes), as previously described in reference to FIGS.8-9. In various implementations, processing component 110 may be adaptedto sense control input signals from control panel unit 1000 and respondto any sensed control input signals received from control knobs 1080,1090. Processing component 110 may be adapted to interpret the controlinput signals as values. In various other implementations, it should beappreciated that control panel unit 1000 may be adapted to include oneor more other control knobs or push buttons (not shown) to providevarious other control functions of infrared imaging system 100A, 100B,such as auto-focus, menu enable and selection, field of view (FoV),and/or various other features. In another embodiment, control panel unit1000 may comprise a single control knob to select image display modes1-3 of FIGS. 8-9.

In another embodiment, control panel unit 1000 may be adapted to beintegrated as part of display component 140 to function as both a userinput device and a display device, such as, e.g., a user activated touchscreen device adapted to receive input signals from a user touchingdifferent parts of the display screen. As such, the GUI interface devicemay have one or more images of, e.g., control knobs 1080, 1090 adaptedto interface with a user and receive user input control values via thetouch screen of display component 140.

In one embodiment, referring to FIG. 10, a first control knob 1080 maybe enabled to select at least one of enhanced vision mode of operation,such as mode 1 (i.e., visible mode only), mode 2 (i.e., infrared modeonly), and mode 3 (blended mode). For example, referring to FIG. 8, thefirst control knob 1080 provides the same scope and features of the userinput control 820. In another embodiment, a single push button forcontrol panel unit 1000 may be used to toggle between each of theenhanced display modes 1-3, without departing from the scope of thepresent disclosure.

In one embodiment, referring to FIG. 10, a second control knob 1090 maybe enabled to select a value from 0-1 for the parameter ξ821, asdescribed in reference to FIG. 8. As previously described, each value ofξ821 (i.e., from 0-1) is adapted to produce a blended image for displaywhere the luminance part (Y) of the blended image includes informationfrom both the visible and infrared spectrum signals.

In maritime applications, embodiments of the present disclosure havesome valuable uses. For example, when navigating at night, anelectro-optical (E/O) camera may typically only be able to image, or maybe managed to only image, a few point sources of light, such as buoys,lights on other boats, lights on land, or lighthouses. With no otherreference, it may be difficult to determine which of these sources thelight belongs. The thermal imager may be adapted to clearly show buoys,islands, other boats, etc., but may not show the color of any lights.Looking at either of the two sensor outputs on a display, an operator ofthe boat may lack essential information for navigation. For example, inone aspect, it may be difficult to determine whether a light is visiblein the video from the E/O imager is from a source on land or from abuoy. In another aspect, given knowledge that a point source isproviding maritime navigation guidance, reliance solely on limited E/Oradiance output may be insufficient for safe navigation. Similarly, inthe video from the thermal imager, several buoys might be clearlyvisible but the lack of color information and can make it hard todetermine the appropriate path to navigate. Fusing information from theE/O and thermal imager as described herein provides the operator of thevessel with all necessary information in a single video stream.Specifically, correlation of the output between the visible-band andinfrared-band cameras is performed for the user such that the operatormay be able to tell which light belongs to a source on land (e.g., acar) and which light belongs to a buoy.

In an enhanced vision system (EVS) for aircraft, the video blendingsystem described herein is an important navigation aid. When attemptingto land at night (under reasonable conditions for thermal imaging), athermal camera provides the operator of the aircraft with an image of anairfield and its surroundings. For rural airports, the thermal imagermay detect obstacles, such as vehicles, animals, and/or people on arunway. These objects may be visible even in poor or zero lightconditions. An imager sensitive to light in the visible light spectrum(E/O imager) may only (under conditions described above) pick up visiblelight sources, such as runway lights, warning lights, street lights,and/or lights on other planes or vehicles. A color E/O imager may beable to, e.g., distinguish red lights from green or white lights, whichmay in some situations be of importance to the operator of the aircraft.During takeoff, landing, or while taxiing, the pilot wants minimaldistraction. Looking at two separate video streams and trying to figureout which light, imaged by the E/O system, go with which structure,imaged by the thermal system, can be a distraction. The combining visionsystem strives to present a pre-correlated blend of critical sceneinformation, e.g., colored light sources for navigation and thermalradiance for situational awareness to support safe navigation withreduced operator distraction. In this case, a single video streamshowing color information from the E/O video merged with the thermalvideo from a thermal imager is advantageous and gives the pilot maximuminformation with minimum distraction.

Driving at night in suburban or rural areas limits driver vision tosections lit up by headlights and/or street lights. Thermal camerasprovide situational overview, and thermal cameras have the addedadvantage of not being “blinded” by head beams from oncoming traffic. AnE/O camera is typically able to image traffic signs, traffic lights,license plates, vehicle warning lights, and road markings with greaterdetail and contrast than the thermal imager. However, in accordance withembodiments of the present disclosure, a fused or blended image providesdrivers, passengers, and/or remote operators with more information thanany one of the video streams alone as a single output instead ofrequiring the viewer to correlate the output between the separatedisplay of two-band camera systems. In an automatic collision avoidancesystem, the added information from the two blended wave bands may alsoprovide better accuracy (e.g., lower false pedestrian detection rates).

In a handheld system with both E/O and thermal imagers, the videoblending scheme described herein is useful in a number of scenarios.When operating at night, colored lights may appear in a thermalrepresentation of the scene when operating in blended mode. Areas lit upby, for example, a flashlight may also benefit from the colorinformation captured by the E/O imager. Specifically, it may be easy fora user to see which area is lit up by looking at the blended videostream and clearly seeing the light beam from the flash light inreference to the entire scene, which may be visible only in the thermalspectrum.

In daytime operation, best situational overview may be achieved using anE/O imager due to retention of color information, which closelyresembles what is seen with the human eye. When looking for somethingwith a thermal contrast, such as a person, animal, fluid spill, etc.,the thermal imager may provide a better chance of detection. Moreover,in bright daytime scenes, the thermal imager may provide radianceinformation in shadowed regions that would be lost to an E/O imagerhandling a wide daytime dynamic range. As such, merging the wave bandsas described herein may provide the user with a color overview of thescene with added contrast for objects with thermal signatures makingsuch a tool useful in a number of applications including lawenforcement, inspection, and/or personal use.

Where applicable, various embodiments of the invention may beimplemented using hardware, software, or various combinations ofhardware and software. Where applicable, various hardware componentsand/or software components set forth herein may be combined intocomposite components comprising software, hardware, and/or both withoutdeparting from the scope and functionality of the present disclosure.Where applicable, various hardware components and/or software componentsset forth herein may be separated into subcomponents having software,hardware, and/or both without departing from the scope and functionalityof the present disclosure. Where applicable, it is contemplated thatsoftware components may be implemented as hardware components andvice-versa.

Software, in accordance with the present disclosure, such as programcode and/or data, may be stored on one or more computer readablemediums. It is also contemplated that software identified herein may beimplemented using one or more general purpose or specific purposecomputers and/or computer systems, networked and/or otherwise. Whereapplicable, ordering of various steps described herein may be changed,combined into composite steps, and/or separated into sub-steps toprovide features described herein.

In various embodiments, software for mode modules 112A-112N may beembedded (i.e., hard-coded) in processing component 110 or stored onmemory component 120 for access and execution by processing component110. As previously described, the code (i.e., software and/or hardware)for mode modules 112A-112N define, in one embodiment, preset displayfunctions that allow processing component 100A, 100B to switch betweenthe one or more processing techniques, as described in reference toFIGS. 2-4 and 8-9, for displaying captured and/or processed infraredimages on display component 140.

Embodiments described above illustrate but do not limit the disclosure.It should also be understood that numerous modifications and variationsare possible in accordance with the principles of the presentdisclosure. Accordingly, the scope of the disclosure is defined only bythe following claims.

What is claimed is:
 1. A method comprising: receiving visible lightsensor data corresponding to a visible light image; receiving infraredsensor data corresponding to an infrared image comprising thermalinformation from an infrared wavelength spectrum; receiving a controlparameter; scaling a luminance data part of the visible light sensordata by a first amount using the control parameter; scaling a luminancedata part of the infrared sensor data by a second amount using thecontrol parameter; and generating output image data by combining thescaled luminance data parts.
 2. The method of claim 1, furthercomprising adjusting the received control parameter prior to the scalingof at least one of the luminance data parts.
 3. The method of claim 1,wherein the visible light image and the infrared image comprise aplurality of pixels, the method further comprising selectively scalingat least one of the luminance data parts on a per pixel basis.
 4. Themethod of claim 3, wherein the selectively scaling comprises:determining a color measurement associated with each pixel of thevisible light sensor data; and selectively increasing the luminance dataof each pixel of the visible light sensor data based on the deter minedcolor measurement associated with the pixel.
 5. The method of claim 4,wherein the selective increase is performed for each pixel having acolor measurement above a threshold.
 6. The method of claim 4, whereinthe amount of the selective increase for each pixel is proportional tothe color measurement.
 7. The method of claim 4, wherein the selectiveincrease in luminance and the color measurement have a nonlinearrelationship.
 8. The method of claim 1, wherein the control parameter isa user selected control parameter, the method further comprising:receiving a control signal identifying one of a plurality of selectableprocessing modes including at least one of a night docking mode, a manoverboard bode, a night cruising mode, a day cruising mode, a hazyconditions mode, or a shoreline mode; and processing the visible lightsensor data and the infrared sensor data in accordance with the selectedprocessing mode.
 9. The method of claim 1, wherein the generatingfurther comprises merging a chrominance data part of the visible lightsensor data with the scaled luminance data parts, the method furthercomprising displaying an output image based on the output image data.10. A non-transitory computer-readable medium on which is storedinformation for performing the method of claim
 1. 11. A systemcomprising a processing component adapted to perform the method ofclaim
 1. 12. A system comprising: a visible light sensor adapted tocapture a visible light image; an infrared sensor adapted to capture aninfrared image comprising thermal information from an infraredwavelength spectrum; a control component adapted to receive a user inputcorresponding to a user selected control parameter; and a processingcomponent adapted to: receive visible light sensor data corresponding tothe visible light image, receive infrared sensor data corresponding tothe infrared image, receive the user selected control parameter, scale aluminance data part of the visible light sensor data by a first amountusing the control parameter, scale a luminance data part of the infraredsensor data by a second amount using the control parameter, and generateoutput image data by combining the scaled luminance data parts.
 13. Thesystem of claim 12, wherein the processing component is further adaptedto adjust the received control parameter prior to the scaling of atleast one of the luminance data parts.
 14. The system of claim 12,wherein the visible light image and the infrared image comprise aplurality of pixels, wherein the processing component is further adaptedto selectively scale at least one of the luminance data parts on a perpixel basis.
 15. The system of claim 14, wherein the processor isfurther adapted to: determine a color measurement associated with eachpixel of the visible light sensor data; and selectively increase theluminance data of each pixel of the visible light sensor data based onthe determined color measurement associated with the pixel to performthe selective scaling.
 16. The system of claim 15, wherein the processoris further adapted to perform the selective increase for each pixel ofthe visible light sensor data having a color measurement above athreshold.
 17. The system of claim 15, wherein the amount of theselective increase for each pixel is proportional to the colormeasurement.
 18. The system of claim 15, wherein the selective increasein luminance and the color measurement have a nonlinear relationship.19. The system of claim 12, wherein the control parameter is a userselected control parameter, wherein the processor is further adapted to:receive a control signal identifying one of a plurality of selectableprocessing modes including at least one of a night docking mode, a manoverboard bode, a night cruising mode, a day cruising mode, a hazyconditions mode, or a shoreline mode; and process the visible lightsensor data and the infrared sensor data in accordance with the selectedprocessing mode.
 20. The system of claim 12, wherein the processingcomponent is further adapted to merge a chrominance data part of thevisible light sensor data with the scaled luminance data parts togenerate the output image data, the system further comprising a displaycomponent adapted to display an output image based on the output imagedata.